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This study examined the longitudinal effects of a middle school reform mathematics 
curriculum on students’ open-ended problem solving in high school. Using assessment 
data from a large, longitudinal project, we compared the open-ended problem-solving 
performance and strategy use of high school students who had used the Connected 
Mathematics Program (CMP) in middle school with that of students who had used 
more traditional mathematics curricula. When controlling for sixth-grade state 
mathematics test performance, high school students who had used CMP in middle 
school had significantly higher scores on a multipart open-ended problem. In addition, 
high school students who had used CMP appeared to have greater success 
algebraically abstracting the relationship in the task. 


INTRODUCTION 


Problem solving is an integral focus of the school mathematics curriculum. Studies of 
problem solving in mathematics education have already moved from a focus only on 
the product (1.e., the actual solution) to a focus on the process (1.e., the set of planning 
and executing activities that direct the search for solution). Individual differences in 
solving mathematical problems can sometimes be understood in terms of differences in 
the uses of different strategies. Proficiency in solving mathematical problems is 
dependent on the acquisition, selection, and application of both domain-specific 
strategies and general cognitive strategies (Schoenfeld, 1992; Simon, 1979). Thus, 
competence in using appropriate problem-solving strategies reflects students’ degrees 
of performance proficiency in mathematics. This implies that assessment tasks should 
reveal the various strategies that students employ. In addition, students’ 
problem-solving strategies become more effective over time. In fact, researchers have 
long used the examination of problem-solving strategies to assess and evaluate 
instructional programs and education systems (Cai, 1995; Fennema et al., 1998). 
Therefore, both the examination of the strategies that students apply and the success of 
those applications can provide information regarding the developmental status of 
students’ mathematical thinking and reasoning. 


The purpose of this study is to use problem solving strategies to investigate how the use 
of different types of middle school curricula affects the learning of high school 
mathematics for a large sample of students from ten high schools in an urban school 
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district. This paper reports findings from a large project, Longitudinal Investigation of 
the Effect of Curriculum on Algebra Learning (LieCal). 


BACKGROUND AND RATIONALE OF THE STUDY 


The LieCal Project began with an investigation of the differential effects of a reform 
middle school mathematics curriculum called the Connected Mathematics Program 
(CMP) and more traditional (called non-CMP) curricula on middle school students’ 
learning of algebra. The CMP and non-CMP curricula are very different. In particular, 
they make use of strikingly different conceptions about algebra — a functional approach 
in the CMP curriculum and a structural approach in the non-CMP curricula. For 
example, the CMP curriculum defines a variable as a quantity that changes or varies. 
The variable idea is needed to describe relationships in the problem situations that the 
CMP curriculum uses. In contrast, the non-CMP curricula define a variable as a 
symbol (or letter) used to represent a number. Variables are treated predominantly as 
placeholders and are used to represent unknowns in expressions and equations. By 
introducing the concept of variables in this fashion, the non-CMP curricula support a 
structural approach to algebra. In the non-CMP curricula, similarly, equation solving 
is introduced symbolically by using the additive and multiplicative properties of 
equality (equality is maintained if the same quantity is added to, subtracted from, 
multiplied by, or divided into both sides of an equation). On the other hand, in the CMP 
curriculum, equation solving is introduced using real-life contexts that are 
incorporated into contextually based justifications of the equation-solving steps. 


In the LieCal Project, we found that on open-ended tasks assessing conceptual 
understanding and problem solving, the growth rate for CMP students over the three 
middle school years was significantly greater than that for non-CMP students (Cai et 
al., 2011). At the same time, CMP and non-CMP students showed similar growth over 
the three middle school years on the multiple-choice tasks assessing computation and 
equation-solving skills. These findings suggest that the use of the CMP curriculum is 
associated with a significantly greater gain in conceptual understanding and problem 
solving than is associated with the use of the non-CMP curricula. However, those 
relatively greater conceptual gains do not come at the cost of lower basic skills, as 
evidenced by the comparable results attained by CMP and non-CMP students on the 
computation and equation solving tasks. 


The LieCal Project has subsequently followed the students into their high school years. 
All high schools in the district are required to use the same district-adopted 
mathematics curriculum. CMP and non-CMP students were mixed into each class in 
each of ten high schools in the same district. Thus, all of the former CMP and 
non-CMP students used the same curriculum and were taught by the same teachers in 
their high schools. We have been examining whether the superior problem-solving 
abilities gained by the CMP students in middle school result in better performance on a 
delayed assessment of mathematical problem solving in high school. 
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In a previous study, we used problem posing as a measure of middle school curricular 
effect on students' learning in high school (Cai et al., 2013). Using problem posing as a 
measure, we found that in high school, students who had used the CMP curriculum in 
middle school performed equally well or better than students who had used more 
traditional curricula. The findings from this previous study not only showed evidence 
of the strengths one might expect of students who used the CMP curriculum, but also 
demonstrated the usefulness of employing a qualitative rubric to assess different 
characteristics of students’ responses to the posing tasks. In the same vein, the present 
study uses open-ended problem-solving strategies as a measure to examine 
longitudinal curricular effect on students’ learning. 


METHOD 
Participants 


In the LieCal Project, we followed more than 1,300 students (650 using CMP and 650 
using non-CMP curricula) from a school district in the United States for three years as 
they progressed through grades 6-8. In the 2008-2009 school year, most of these 1,300 
CMP and non-CMP students from the middle school study entered high schools as 
freshmen. We then followed the students enrolled in the 10 high schools that have the 
largest numbers of the original 1,300 CMP and non-CMP students. 


Assessment Tasks and Analyses 


As part of the LieCal Project, we developed and used 13 open-ended tasks to assess 
students’ learning in high school, specifically the 11th and 12th grades. Students’ 
responses were analyzed in two ways. The first was to quantitatively score each 
student response using a prior-developed holistic scoring rubric. The second was to 
qualitatively analyze students’ responses with a focus on their solution strategies. In 
this paper, we mainly draw on results from an analysis of solution strategies to a 
pattern problem called the doorbell problem (see Appendix). This five-part task 
assesses students’ ability to find regularities of a pattern and make generalizations. We 
chose to report the results from this task as it is a representative task that assesses 
students’ generalization skills. 


Data Collection and Coding 


As part of the larger longitudinal study, we assessed students in the fall of 11™ grade 
(Fall, 2010), spring of 11" grade (Spring 2011), and spring of 12" grade (Spring 2012). 
The data for the analyses of students’ strategies came mainly from the 12th grade 
spring assessment. In a small number of cases, if a student did not participate in the 
Spring 2012 assessment but did participate in the Spring 2011 assessment, we used the 
data from the Spring 2011 assessment. If a student did not participate in either the 
Spring 2012 or Spring 2011 assessments, but had participated in the Fall 2010 
assessment, we used the data from the Fall 2010 assessment. This allowed us to look at 
the students’ most recent attempt at each task. 
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As noted above, students’ responses to the doorbell problem were first scored using a 
holistic scoring rubric that took into account the students’ numerical answers and their 
explanations of their strategies. The responses were then also qualitatively coded for 
the types of strategies used. We coded students’ solution strategies for parts A, B, C, 
and E as an abstract strategy, a concrete strategy, an unidentifiable strategy, or no 
strategy. Students who used an abstract strategy were able to recognize that the number 
of guests entering for each ring was equal to either two times the ring number minus 
one (1.e., y = 2n — 1) or the ring number plus the ring number minus one (i.e., y= + (n 
— 1)). Students who used a concrete strategy were able to identify that the number of 
guests who enter increases by two for each doorbell ring and then sequentially adding 
two until they reached the desired number of rings, but did not abstract an algebraic 
formula. An unidentified solution strategy was a strategy that did not particularly make 
sense for the problem (e.g., y = [7(100) + 2] — 1). Lastly, a student was said to have used 
no strategy if the student did not show work for his or her answer, or if he or she did not 
attempt to answer the question at all. 


Students’ strategies for part D were coded in one of five ways. First, the student could 
have completely abstracted the algebraic formulas 2 — 1 orn +(n—1). Secondly, they 
could have completely abstracted the pattern in a verbal description (e.g. “The number 
of guests who entered on a particular ring of the doorbell equalled two times that ring 
number minus one.”). Third was an incomplete abstraction that only captured a 
recursive relationship, such as, “When the bell rings, two more people come.” Fourth 
was an unidentified strategy, which either represented the strategies for students who 
incorrectly answered the question or had a provided a strategy that did not make sense. 
Finally, a strategy was coded as “no strategy” if no attempt was made to solve the 
problem. 


RESULTS 
Overall Performance on the Doorbell Problem 


We first conducted two ANCOVA analyses based on the quantitative scoring to 
student responses to the doorbell problem. The ANCOVA analyses indicated 
significant curriculum effects under two covariates for the doorbell problem. When 
controlling for overall state math test exam scores for 6th grade, CMP students scored 
significantly higher than non-CMP students on the doorbell problem (t¢ = 2.09, p = 
0.0371). When controlling for scores on the algebra subtest on the overall state math 
test for 6th grade, CMP students still scored significantly higher than non-CMP 
students (t = 2.47, p = 0.0141). 


Performance on Individual Parts of the Doorbell Problem 


Chi-squared tests were performed to look for relationships between curriculum and 
correctness of answers on each part of the doorbell problem. For part A, there was a 
significant relationship between curriculum and correct answers (7° = 6.5363, p < 
0.040). That is, a significantly larger percentage of the CMP students had correct 
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answers than the non-CMP students. For parts B, C, D, and E, there were no significant 
relationships between curriculum and correct answers. For each of the five parts of the 
problem, Table 1 provides the percentage of students with correct answers in that part. 
Note that Table 1 shows a considerable decreasing trend in the number of students who 
found a correct solution from part A to part E. 


Doorbell Problem Part 
Curriculum A B C D E 
CMP (n = 321) 80.1 38.6 21d 18.1 1D 
Non-CMP (n = 212) 74.5 35.4 27.4 16.0 D2 


Table 1: Percentages of CMP and non-CMP students who correctly solved each part of 
the Doorbell Problem 


Concrete and Abstract Solution Strategies 


Focusing specifically on the solution strategies of those students who provided correct 
solutions for parts of the Doorbell problem, some differences in strategy use arose 
between the two groups. For part B (see Table 2), 73.4% of CMP students (n=124) and 
60% of non-CMP students (n=75) abstracted the problem to an algebraic formula to 
find the correct solution, whereas 17.7% of CMP students and 24.0% of non-CMP 
students used a concrete strategy. A significantly greater proportion of CMP students 
used the abstract strategy than did the non-CMP students (z = 1.97, p < 0.050), but 
there was no significant difference in proportion between CMP and non-CMP students 
for the concrete strategy. 


For part C (see Table 2), 71.9% of CMP students (n=89) and 67.2% of non-CMP 
students (n=58) abstracted the problem to an algebraic formula, whereas 7.9% of CMP 
students and 19.0% of non-CMP students used concrete strategies to find a correct 
solution. A significantly greater proportion of non-CMP students used the concrete 
strategy than did the CMP students (z = -2.27, p < 0.025), but there was no significant 
difference in proportion between CMP and non-CMP students for the abstract strategy. 


For part A (see Table 2), 67.3% of CMP students (7 = 257) and 63.9% of non-CMP 
students (n = 158) used a concrete strategy to find the correct answer, whereas 26.1% 
of CMP students and 27.8% of non-CMP students abstracted the problem to an 
algebraic formula. There were no significant differences in proportion between CMP 
and non-CMP students for each strategy. 


For part D, almost every student who provided a correct solution responded in nearly 
the same way. All of the 34 non-CMP students and 54 out of 58 CMP students who 
correctly answered this part generated an algebraic abstraction and provided a 
mathematical formula. The remaining four CMP students wrote out a verbal 
description of the mathematical formula, which would still require them to have first 
abstracted the relationships before translating those relationships into written form. 
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Type of strategy 

Problem part n Abstract Concrete Unidentified None 
A 

CMP Zot 26.1 67.3 i 3.1 

Non-CMP 158 27.8 63.9 1.9 6.3 
B 

CMP 124 73.4 it a2 5.6 

Non-CMP fs) 60.0 24.0 4.0 12.0 
C 

CMP 58 71.9 7.9 9.0 L12 

Non-CMP 34 67.2 19.0 a2 8.6 
D 

CMP 58 100.0 0.0 0.0 0.0 

Non-CMP 34 100.0 0.0 0.0 0.0 
E 

CMP 24 62.5 29.2 4.2 4.2 

Non-CMP 11 45.5 36.4 0.0 18.2 


Table 2: Percentages of CMP and non-CMP students who used each type of strategy to 
correctly answer parts of the doorbell problem 


Part E seemed to be a challenging question for both the CMP and non-CMP students. 
Only 24 CMP students and 11 non-CMP students provided a correct solution to this 
part of the doorbell problem. Given these small sample sizes, although there were 
noticeable group differences in raw percentages of students using algebraic and 
concrete strategies, with a greater proportion of CMP students than of non-CMP 
students using algebraic strategies, these differences were not statistically significant. 


DISCUSSION 


As part of a larger longitudinal study of curricular effect on mathematics learning, the 
results we have presented above provide a useful perspective on the potential 
long-term impacts of reform mathematics curricula on students’ mathematical thinking 
and problem solving. Although we have presented data from only one open-ended 
task, the results suggest that high school students who used the CMP curriculum in 
middle school were more successful than their peers who used more traditional 
middle-school curricula at solving the doorbell problem and explaining their solution 
strategies. This result accords with those obtained when these students were still in 
middle school (Cai, et al., 2011). The result is also consistent with our previous 
findings using problem posing as measure of curricular effect (Cai et al., 2013). Thus, 
it would appear that the CMP students’ problem-solving gains persist well into high 
school. 
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The retention of these gains over longer time intervals also parallels the findings from 
research on the effectiveness of problem-based learning (PBL) in medical education 
(Hmelo-Silver, 2004). In that context, medical students trained using a PBL approach 
performed better than non-PBL students on conceptual understanding and 
problem-solving ability even when assessed at a later time. In a similar fashion, the 
CMP students in the LieCal project experienced problem-based instruction that 
focused on developing students’ conceptual understanding and problem solving 
abilities. 

In addition, our analysis of the strategies used by the students in this study suggests that 
the CMP students who correctly solved the parts of the doorbell problem were 
somewhat more likely to make generalizations. This appears to reflect the emphasis in 
the CMP curriculum on relationships between quantities (1.e., the functional approach). 
The ability to abstract algebraic relationships from real-world situations appears to also 
have persisted in the CMP students. 


Note that for this analysis, we focused on the strategies of students who correctly 
answered one or more parts of the doorbell problem. We did not consider the strategies 
of students who failed to provide correct answers. Additional analyses that will further 
probe the strategies of students who provided incorrect answers are in progress at the 
time of this proposal. Also, we are analysing data from other open-ended problems. 


APPENDIX 

Sally is having a party. 
The first time the doorbell rings, 1 guest enters. 
The second time the doorbell rings, 3 guests enter. 
The third time the doorbell rings, 5 guests enter. 
The fourth time the doorbell rings, 7 guests enter. 


Keep going in the same way. On the next ring a group enters that has 2 more persons 
than the group that entered on the previous ring. 


A. How many guests will enter on the 10" ring? Explain or show how you found 
your answer. 


B. How many guests will enter on the 100" ring? Explain or show how you found 
your answer. 


C. 299 guests entered on one of the rings. What ring was it? Explain or show how 
you found your answer. 


D. How many guests will enter on the n™ ring? Show or explain how you found your 
answer. 


E. If we count all of the guests who entered on the first 100 rings, how many would 
we get in total? Show or explain how you found your answer. 
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Note 


The research reported here is supported by a grant from the National Science 
Foundation (ESI-0454739 and DRL-1008536). Any opinions expressed herein are 
those of the authors and do not necessarily represent the views of the National Science 
Foundation. Any opinions expressed herein are those of the authors and do not 
necessarily represent the views of the National Science Foundation. 
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